Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

adding AI project #944

Merged
merged 17 commits into from
Jul 29, 2022
Merged

adding AI project #944

merged 17 commits into from
Jul 29, 2022

Conversation

brettsam
Copy link
Member

@brettsam brettsam commented Jun 23, 2022

This PR introduces a new nuget package (in preview): Microsoft.Azure.Functions.Worker.ApplicationInsights. This package should help to address issues like #702, #760, and others.

The goal of this new package is to simplify the Application Insights wire-up required in order to have proper correlation between the RequestTelemetry generated in the Functions host and any telemetry generated during the invocation on the isolated worker.

The package uses as much of the default Application Insights setup as possible so that we can utilize the existing documentation and (hopefully) not introduce many new Functions-only concepts.

Background

Azure Functions has automatic support for App Insights built into the host (and for in-proc .NET functions). This emits RequestTelemetry for every invocation, TraceTelemetry for ILogger logs, has Live Metrics enabled and auto-tracks Dependencies like outgoing HTTP calls, SQL queries, etc.

In the out-of-proc worker model, a gRPC connection is made between the worker and host. This connection is used to send invocation details from the host to the worker, but it’s also used for a worker to send log messages back to the host. This allows a worker to expose some logger class that behind-the-scenes sends log messages to the host via gRPC, which then logs it to the configured Application Insights resource through the host. This is the extent of "Application Insights support" that we offer to all of the out-of-proc workers. Side note that this support only works during an invocation. Any logs emitted outside the scope of an invocation are ignored by the host, which results in issues like #702.

For the dotnet-isolated model we want to expand this support to create a richer offering that allows the worker to emit telemetry directly to Application Insights while maintaining the proper telemetry correlation (i.e. Dependencies are auto-tracked and show up as part of the overall Request).

The package introduces two new extension methods available on IFunctionsWorkerApplicationBuilder that allow you register Application Insights services (AddApplicationInsights())and the Application Insights logger (AddApplicationInsightsLogger()).

var host = new HostBuilder()
    .ConfigureFunctionsWorkerDefaults(builder =>
    {
        builder
            .AddApplicationInsights()
            .AddApplicationInsightsLogger();
    })
    .Build();

AddApplicationInsights()

This method internally calls the Application Insights IServiceCollection extension method AddApplicationInsightsTelemetryWorkerService(), with a couple of extra services added to provide the correlation needed for Functions. See the Application Insights documentation for Worker Services for details. The Functions extension method includes an optional callback parameter to allow you to configure the underlying ApplicationInsightsServiceOptions options.

With the default configuration, this will include:

  • Automatic dependency tracking as a child of the parent invocation (which is still tracked via the host).
  • A new per-invocation tracked dependency -- currently of type Azure.Functions with name Invoke that tracks the time spent on the isolated worker.
  • Live Metrics from the worker (as well as the host).

AddApplicationInsightsLogger()

This method internally calls the Application Insights ILoggingBuilder extension method AddApplicationInsights(), with a couple of extra services added to provide the correlation needed for Functions. See the Application Insights documentation for using ILogger in a Console application for details. The Functions extension method includes an optional callback parameter to allow you to configure the underlying ApplicationInsightsLoggerOptions options.

Note that calling this function will disable the gRPC logging that occurs by default; all ILogger logs will be emitted directly to Application Insights as TraceTelemetry. This should address issues like #702 without changing host behavior and affecting other language workers.

Configuration

Some notes on configuration:

  • Using the app setting APPLICATIONINSIGHTS_CONNECTION_STRING will automatically wire up the both the host and the worker to the same Appliation Insights instance.
  • Logging and Application Insights configurations in the worker are completely separate from those configured in host.json. For example, if you've disabled sampling in the host, you will need to also disable it in the worker. Same with configuring logging levels. Please refer to the documentation links above to configure your worker.

Copy link
Member

@fabiocav fabiocav left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very small nit comments

/// <summary>
/// Gets or sets a value indicating whether to send <see cref="ILogger"/> logs through the Functions host.
/// </summary>
public bool DisableHostLogger { get; set; } = false;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Struggling with this becoming a public property on the worker options, as the host logger is a gRPC specific concept, but I really don't have a great alternative at the moment.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One of the things I had previously mentioned in discussion would be to introduce a proper service for user logs. By default, the gRPC implementation would register one (overridable) that would push to the host. If the implementation here registers one as well, that would bypass the gRPC logic without requiring a flag. Thoughts?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me look at that -- will submit a change here soon with this split out.

Copy link
Member

@liliankasem liliankasem left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm! Just 1 question about custom metrics and use of a const file :D

@@ -0,0 +1,94 @@
using System;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit - Are there certain files that need the copyright header and some that don't?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No they all should... will go through and double-chedk!

namespace Microsoft.Azure.Functions.Worker.Logging
{
/// <summary>
/// Minimalistic LogWriter that does nothing.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you elaborate on this LogWriter? Trying to understand it. Is it for when a user doesn't create/define other specific loggers (UserLogWriter, SystemLogWriter) and this is the default put in their place?

This summary caught me off guard 🤣

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All of our "logging through the host" goes through Grpc, which isn't part of the core packages. So we need some default way to basically "do nothing" without breaking dependency injection if for some reason Grpc isn't wired up. In reality this will be very rare.

return async context =>
{
var middleware = context.InstanceServices.GetRequiredService<FunctionActivitySourceMiddleware>();
await middleware.Invoke(context, next);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible that the customer can register a middleware before calling the AddApplicationInsights method in their Program.cs, which will cause the this middleware to be executed only after the customer middleware and any logging coming from customer m/w will miss the code from FunctionActivitySource.StartInvoke? I am guessing when we move this to Core project eventually, that may not be a problem as we can register this m/w before customer m/w?

Copy link
Member Author

@brettsam brettsam Jul 25, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that's why we'll move this behavior to the Core bits before releasing. It won't actually be middleware in the end, we'll put it directly into the FunctionsApplication.InvokeFunctionAsync() so that it's (effectively) un-overrideable.

{
if (WorkerMessage.IsSystemLog)
{
_systemLogWriter.WriteSystemLog(_scopeProvider, _category, logLevel, eventId, state, exception, formatter);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would this log anything ? Because ISystemLogWriter implementation registered to DI container is NullLogWriter which does nothing.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't we ideally call some APIs on TelemetryClient to send this to AI?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This gets overridden by the call to AddGrpc (it adds the GrpcFunctionsHostLogWriter via DI).

System logs are meant to be only for us (in kusto). They have to go back through the host and don't show up in App Insights today. That's why I had to split things out so weirdly -- we needed ways to stop one type of log (user logs) from going through the host, while allowing others (the system logs) to continue flowing.

services.AddSingleton<IUserLogWriter>(p => p.GetRequiredService<GrpcFunctionsHostLogWriter>());
services.AddSingleton<ISystemLogWriter>(p => p.GetRequiredService<GrpcFunctionsHostLogWriter>());
services.AddSingleton<IUserMetricWriter>(p => p.GetRequiredService<GrpcFunctionsHostLogWriter>());
services.AddSingleton<IWorkerDiagnostics, GrpcWorkerDiagnostics>();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unrelated to changes from this PR:
Are we supposed to call _outputChannel.TryWrite(message); here?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Certainly seems like it... I can follow up on that with a separate PR since it's unrelated.

builder.Services.TryAddEnumerable(new ServiceDescriptor(typeof(ITelemetryInitializer), typeof(FunctionsTelemetryInitializer), ServiceLifetime.Singleton));
builder.Services.TryAddEnumerable(new ServiceDescriptor(typeof(ITelemetryModule), typeof(FunctionsTelemetryModule), ServiceLifetime.Singleton));

// User logs will be written directly to Application Insights; this prevents duplicate logging.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We also register GrpcFunctionsHostLogWriter in DotNetWorker.Grpc/GrpcServiceCollectionExtensions.cs. So that is going to send logs to host (duplicate). Correct?

.ConfigureFunctionsWorkerDefaults(builder =>
{
builder
.AddApplicationInsights()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we consider exposing a single method (which internally calls these 2) for the convenience of user?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ideally yes... but App Insights has these as two separate concepts. I'd like to try to mirror their usage as much as possible so we can offload the documentation to them. I agree it's kinda odd... but we can review before making this GA.

Copy link
Member

@fabiocav fabiocav left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's get this in so we can start validating the functionality. We'll continue to iterate as you go. :shipit:

@michaelelleby
Copy link

@brettsam You need to add a dependency on Microsoft.Azure.Functions.Worker version 1.8.0 or higher, as this code crashes when using Microsoft.Azure.Functions.Worker version 1.6.0 with the following exception
System.MissingMethodException: 'Method not found: 'Microsoft.Extensions.DependencyInjection.IServiceCollection Microsoft.Extensions.DependencyInjection.ServiceCollectionExtensions.RegisterDefaultConverters(Microsoft.Extensions.DependencyInjection.IServiceCollection)'.'

@Done3319
Copy link

Done3319 commented Nov 9, 2022

For me this works in general but I still have difficulties getting logs from the isolated worker with a level lower than Warning.
I understand that this now needs to be controlled separately from host.json and therefore tried to controll via appsettings.json:

           var host = new HostBuilder()
                .ConfigureAppConfiguration(c =>
                {
                    c.AddJsonFile("appsettings.json", optional: true, reloadOnChange: true);
                    c.AddJsonFile("local.settings.json", optional: true, reloadOnChange: true);
                    c.AddEnvironmentVariables();
                    c.Build();
                })
                .ConfigureFunctionsWorkerDefaults(builder =>
                {
                        builder
                        .AddApplicationInsights()
                        .AddApplicationInsightsLogger();

                })

appsettings.json has the same settings like host.json which work perfectly for the host.

Inside the worker I tried logging to a ILoggerFactory via DI and also via the FunctionContext.GetLogger().

            _logger.LogWarning("LogWarning Test");
            _logger.LogInformation("LogInformation Test");

The first one comes through, the second one not.

@StMarian
Copy link

StMarian commented Nov 18, 2022

@Done3319 try #1123 (comment) (or #1123 (comment))

@nicolashemery
Copy link

Hello,

I was trying to used isolated function to run Availability test.
I had an issue following as Activity.Current was being null if using packages:
-Microsoft.Extensions.Logging.ApplicationInsights 2.21.0
-Microsoft.ApplicationInsights.WorkerService 2.21.0
I tried to make something with functionContext, but unsuccessfully, and finally went to this package and it worked.

But i already new this package and discarded it in the past for other function as i needed scope logs and its hardcoded in src/DotNetWorker.ApplicationInsights/FunctionsApplicationInsightsExtensions.cs
logging.AddApplicationInsights(options => { options.IncludeScopes = false; configureOptions?.Invoke(options); });

My question is then (probably @brettsam) is there a reason why this is harcoded?

@theo-albers
Copy link

Hi @brettsam, is there a reason why TelemetryClient isn't registered as singleton? Doing so would allow us to resolve that dependency and start tracking events. The scenario I see is to track retries and single successful runs for Azure Service Bus triggers.

Right now https://github.com/Azure/azure-functions-dotnet-worker/blob/main/src/DotNetWorker.ApplicationInsights/FunctionsTelemetryModule.cs creates a TelemetryClient. I see no reason why this dependency can't be registered as singleton.

https://github.com/Azure/azure-functions-dotnet-worker/blob/main/src/DotNetWorker.ApplicationInsights/FunctionsApplicationInsightsExtensions.cs alsready registers the module as singleton.

Or are you worried that someone already registered the TelemetryClient and that's why you would like an instance of "your own"?

@brettsam
Copy link
Member Author

TelemetryClient is registered as a singleton by App Insights already -- https://github.com/microsoft/ApplicationInsights-dotnet/blob/main/NETCORE/src/Shared/Extensions/ApplicationInsightsExtensions.cs#L365. You can use it in any class constructor and see for yourself.

We're doing nothing special with App Insights types or registrations. We call AddApplicationInsightsTelemetryWorkerService() and that SDK handles everything. So in general, if it works in some other project, it'll work here.

As for why the module uses its own client -- I was following the patterns of other modules. (https://github.com/microsoft/ApplicationInsights-dotnet/blob/main/WEB/Src/Web/Web/ExceptionTrackingTelemetryModule.cs)

I seem to recall that during testing this, the modules have some circular dependency issues and you can't inject the client there. I think it's b/c the TelemetryClient depends on TelemetryConfiguration, which calls Initialize() on all modules. I just tried it quickly in a test and got a stack overflow. So that's very specific to TelemetryModules and how they work.

@theo-albers
Copy link

Yes, I discovered that this afternoon: once I include and enable the Functions AI package, I can resolve the TelemetryClient.

@SPSCS-Simon
Copy link

@brettsam given this is merged, what are the plane to actually release theMicrosoft.Azure.Functions.Worker.ApplicationInsights package?

@leozin
Copy link

leozin commented Jul 5, 2023

Hi @brettsam ,
Is it still possible to use TelemetryClient instead of logging when sending Telemetry data to Azure in isolated Azure Functions?
Our platform has many submodules that use TelemetryClient without Dependency Injection, which is created through a static initializer, using APPINSIGHTS_INSTRUMENTATIONKEY.
As our new Azure (Service Bus Listeners) Functions are dotnet-isolated, the TelemetryClient doesn't send telemetry data anymore. The content in the traces table lacks some important data, for example, operationName, customDimensions (like Function Invocation Id) and synthetic source, among other things.
If we use logging, then the data is sent correctly, however, if we want to re-implement our telemetry mechanism using logging and DI, it will require a massive overhaul and we would lose some features like metrics etc.
Previously, when using TelemetryClient, it would populate everything automatically (in a non-isolated function), but now it seems that I have to set all these parameters manually, or I'm doing something wrong when configuring Logging.

My Program.cs is defined as:

var host = Host.CreateDefaultBuilder()
.ConfigureFunctionsWorkerDefaults(builder => builder.AddApplicationInsights().AddApplicationInsightsLogger())
.ConfigureServices() // our Singletons
.Build();

await host.RunAsync();

and our packages are:

<PackageReference Include="Microsoft.ApplicationInsights.WorkerService" Version="2.21.0" />
<PackageReference Include="Microsoft.Azure.Functions.Worker.ApplicationInsights" Version="1.0.0-preview4" />
<PackageReference Include="Microsoft.Azure.Functions.Worker.Sdk" Version="1.11.0" />
<PackageReference Include="Newtonsoft.Json" Version="13.0.3" />
<PackageReference Include="System.Net.NameResolution" Version="4.3.0" />
<PackageReference Include="Microsoft.Azure.Functions.Extensions" Version="1.1.0" />
<PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.ServiceBus" Version="5.11.0" />
<PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Http" Version="3.0.13" />
<PackageReference Include="Microsoft.Azure.Functions.Worker" Version="1.16.0" />
<PackageReference Include="Microsoft.NETCore.Targets" Version="5.0.0" />

Would you have an idea if I need to add some extra config or something else?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.